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GOAL  PLANNING  AND  PURSUIT:  MOBILE  APPLICATION  DEVELOPMENT  AND 
EVALUATION 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  is  invested  in  developing  strategies  to  engage  Soldiers  in  self-learning 
and  self-development.  The  Army  Learning  Model  (ALM)  emphasized  learner-centered 
instruction  and  life-long  learning  through  self-development  (Department  of  the  Army,  Training 
and  Doctrine  Command,  2011).  To  assist  Soldiers  in  pursuing  goal-centered  self-learning,  this 
research  developed  mobile  applications  in  order  to  compare  two  goal  planning  and  pursuit 
techniques.  One  mobile  application  used  the  Army’s  current  SMART  (i.e.,  Specific, 
Measureable,  Action-oriented,  Realistic  &  Relevant,  Timely)  goals  approach.  The  other  was 
designed  around  mental  contrasting  and  implementation  intentions,  an  alternative  to  SMART 
goal  planning  techniques  that  has  produced  promising  results  in  traditional  academic  settings. 

Procedure: 

The  researchers  developed  two  goal  setting  and  goal  pursuit  applications  for  hand-held 
devices  and  then  compared  the  effectiveness  of  these  applications  with  Noncommissioned 
Officers  enrolled  in  Advanced  Leaders  Courses  at  Fort  Leonard  Wood,  MO.  The  media  design 
entailed  presenting  the  steps  to  guide  Soldiers  through  the  process  of  mental  contrasting  and 
implementation  intentions  using  a  mobile  application.  The  application  was  developed  in  both 
iOS  (iPhone)  and  Android  versions.  This  application,  entitled  Anny  Achieve,  consisted  of  five 
key  steps  for  identifying  goals,  outcomes,  obstacles,  actions  to  overcome  obstacles  and 
formulation  of  an  if/then  plan  for  goal  pursuit.  The  application  also  included  a  method  for 
tracking  goal  progress  and  a  short  evaluation  questionnaire  to  be  completed  by  participants  when 
evaluating  the  application. 

For  comparison,  an  additional  mobile  application  was  development  based  on  the  Army 
Goal  Book  (Department  of  the  Army,  Anny  Center  for  Enhanced  Performance,  n.d.).  This 
application  used  the  traditional  SMART  goal  planning  and  pursuit  strategy.  The  application 
consisted  of  steps  involving  the  identification  of  a  long-tenn  goal,  key  strengths  and  skills  related 
to  the  goal,  obstacles  to  achieving  the  goal,  priority  for  development,  key  actions  and  beliefs  that 
would  aid  in  reaching  the  goal,  related  short  tenn  goals,  and  a  means  to  keep  the  goal  and  other 
infonnation  in  mind  while  pursuing  the  goal.  The  same  questionnaire  used  in  Anny  Achieve 
was  included  in  the  user  evaluation  for  this  application. 

Noncommissioned  Officers  (NCO)  enrolled  in  Advanced  Leaders  Courses  (ALC)  at  the 
Fort  Leonard  Wood  NCO  Academy  (NCOA)  participated  in  this  research.  The  participants  met 
with  the  researchers  in  a  classroom  at  the  NCOA.  Participants  were  briefed  on  the  experiment 
and  were  able  to  download  the  application  through  the  classroom’s  wireless  network.  NCOs 
who  consented  to  participate  downloaded  and  installed  the  mobile  application  onto  their  phones. 


IV 


Findings: 


The  participants  were  instructed  to  complete  the  basic  goal  setting  requirements  of  their 
application  and  to  return  the  application  evaluation  by  email  at  the  end  of  their  course.  A  total  of 
113  students  participated  in  the  application  orientation  sessions  and  consented  to  participate.  At 
the  end  of  the  evaluation  period,  nine  (9)  evaluations  were  returned.  Due  to  the  small  number  of 
evaluations  returned  by  participants,  no  in-depth  analyses  concerning  the  relative  effectiveness  of 
each  approach  are  presented.  The  limited  data  did  not  indicate  a  measurable  difference  between 
applications  in  terms  of  their  overall  user  evaluation  scores. 

In  the  orientation  sessions,  however,  a  number  of  useful  suggestions  were  made.  The 
evaluation  presented  here  focuses  on  feedback  from  participants,  specifically  related  to  the  need 
to  tie  a  goal  planning  application  to  existing  Army  systems,  such  as  the  Army  Career  Tracker, 
and  allowing  for  more  interactivity  and  feedback  from  the  application  as  far  as  scheduling  of 
goals,  reminders  and  other  infonnation  that  may  aid  in  the  completion  of  the  goals.  Finally, 
lessons  learned  and  recommendations  are  presented  concerning  the  distribution  and  evaluation  of 
mobile  applications  in  an  Army  training  context. 

Utilization  and  Dissemination  of  Findings: 

Results  of  the  research  were  provided  to  the  Institute  for  Noncommissioned  Officer 
Professional  Development.  This  report  describes  the  mobile  applications  development 
procedures  and  provides  detailed  descriptions  of  the  page/screen  content  and  screen  captures  for 
both  applications  developed  for  this  project.  The  principal  utilization  of  this  report  is  for  future 
development  of  mobile  applications  and  recommendations  for  conducting  field  evaluations  of 
mobile  applications.  We  also  include  recommendations  to  potentially  improve  evaluations 
which  require  input  from  Soldiers  over  an  extended  time  period  or  independent  of  an  onsite, 
scheduled  data  collection  period. 
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Goal  Planning  and  Pursuit:  Mobile  Application  Development  and  Evaluation 


Introduction 

The  U.S.  Army  is  invested  in  developing  strategies  to  engage  Soldiers  in  self-learning 
and  self-development.  The  Army  Learning  Model  (ALM)  emphasizes  learner-centered 
instruction  and  life-long  learning  through  self-development  (Department  of  the  Army,  Training 
and  Doctrine  Command,  2011).  To  assist  Soldiers  in  pursuing  goal-centered  self-learning,  this 
research  developed  mobile  applications  in  order  to  compare  two  goal  planning  and  pursuit 
techniques.  One  mobile  application  applied  the  Army’s  current  SMART  (i.e.,  Specific, 
Measureable,  Action-oriented,  Realistic/Relevant,  and  Timely)  goals  approach.  The  other 
application  was  designed  around  mental  contrasting  and  implementation  intentions,  a 
streamlined  alternative  to  SMART  goal  planning  strategies,  which  has  shown  promise  in 
traditional  academic  settings  (Gollwitzer,  Oettingen,  Kirby,  &  Duckworth,  2011). 

Background  on  the  Research  Problem 

Soldiers  face  a  number  of  developmental  responsibilities.  Among  others,  these 
responsibilities  include  learning  and  maintaining  their  technical  skills,  developing  as  a  leader, 
maintaining  their  physical  fitness,  and  adhering  to  Army  values.  With  rapid  deployment  cycles, 
abbreviated  training  opportunities,  and  ambiguous  self-development  requirements,  Soldiers  are 
challenged  to  maintain  the  balance  between  mission  accomplishment  and  their  career-  and  self¬ 
development  needs.  To  address  the  need  for  continuous  learning  and  growth  among  Soldiers,  the 
Anny  is  looking  to  technological  innovations,  such  as  mobile  device  applications,  to  assist  in 
balancing  these  demands. 

The  Anny  Learning  Model  (ALM)  has  focused  on  increased  responsibility  for  Soldiers  to 
leam  on  their  own  in  a  life-long  learning,  professional  development  context  (Department  of  the 
Anny,  Training  and  Doctrine  Command,  2011).  Moreover,  ALM  encourages  the  use  of  digital 
applications  to  provide  learning  opportunities  and  support  Soldiers’  self-guided  learning  needs. 

A  key  aspect  of  this  strategy  is  self-motivation  and  goal  pursuit.  Our  research  focused  on  self- 
motivation  and  goal  pursuit  to  assist  Soldiers  in  identifying  developmental  goals  and  finding 
strategies  to  reach  their  goals  both  within  and  beyond  institutional  training. 

While  the  emphasis  on  the  Soldier’s  role  in  their  own  development  is  apparent,  recent 
surveys  of  Anny  personnel  continue  to  indicate  concern  over  the  time  allocated  for  self¬ 
development  activities  (Steele,  2011).  Self-learning  represents  a  powerful  supplement  to 
institutional  and  unit  training;  however,  it  can  be  challenging  for  Soldiers  to  manage  the  time 
required  for  concentrated  self-directed  learning  (Graves,  Rauchfuss,  &  Wisecarver,  2012). 

One  method  to  address  Soldiers’  expectations  related  to  self-development  is  to  provide 
tools  to  support  their  self-directed  learning  pursuits.  In  fact,  recent  Anny  efforts  have  identified 
methods  for  addressing  these  expectations.  For  example,  the  Connecting  Soldiers  to  Digital 
Applications  project  sought  to  investigate  opportunities  to  integrate  mobile  devices  and 
applications  with  Soldier  administrative,  training,  and  tactical  functions  and  led  to  the 
development  of  a  host  of  training  and  personal  effectiveness  applications  (Dragon  &  McCarthy, 
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2012).  Mobile  applications  of  all  kinds  are  emerging  to  assist  individuals  and  teams  become 
more  efficient  and  disciplined  in  their  personal  and  professional  development. 

Approaches  to  Goal  Planning  and  Pursuit 

Goal  setting  theory  has  been  shown  to  be  an  effective  self-development  strategy  across  a 
range  of  performance  domains  (Locke  &  Latham,  2002).  Setting  specific,  challenging  goals  can 
motivate  a  person  to  achieve  desirable  objectives.  Research  related  to  self-regulation  of  behavior 
change  has  revealed  a  number  of  best  practices  related  to  how  a  person  can  achieve  behavior 
change  by  effectively  pursuing  carefully  selected  goals  (see  Locke  &  Latham,  2002). 

SMART  Goals  Approach.  The  U.S.  Anny  has  embraced  the  value  and  use  of  goal 
setting  in  doctrine  and  practice.  Various  Army  Goal  Books  (Department  of  the  Army,  Pamphlet 
350-1-2,  2010;  Department  of  the  Army,  Army  Center  for  Enhanced  Performance,  n.d.),  include 
goal  planning  and  pursuit  information  as  part  of  the  Comprehensive  Soldier  Fitness  program. 1 
The  goal  planning  and  pursuit  strategies  and  processes  included  in  the  Army  Goal  Books  are 
based  upon  what  is  commonly  known  as  SMART  goal  setting.  SMART  is  an  acronym  that 
represents  the  five  key  characteristics  of  goals  that  have  been  supported  in  the  goal  setting 
literature.  Goals  are  most  likely  to  be  achieved  when  they  are  Specific,  Measurable,  Attainable, 
Realistic  and  Time  bound  (Doran,  1981). 

Mental  Contrasting  and  Implementation  Intentions  Approach.  A  self-regulation 
strategy  that  has  been  shown  to  help  people  wisely  select  and  pursue  goals  is  mental  contrasting 
of  what  they  anticipate  for  the  future  with  their  current  situation  (Oettingen,  2000,  2012; 
Oettingen,  Pak,  &  Schnetter,  2001).  People  who  employ  mental  contrasting  first  imagine  having 
attained  a  desired  future  outcome  (e.g.,  do  more  exercise)  and  then  imagine  what  in  their  present 
situation  impedes  them  in  attaining  their  desired  future  state  (e.g.,  feeling  tired).  When  chances 
of  success  are  high,  mental  contrasting  helps  goal-seekers  pursue  their  goals;  when  chances  of 
success  are  low,  mental  contrasting  helps  goal-seekers  revise  their  goals  in  order  to  pursue  more 
promising  alternatives. 

Various  lines  of  research  have  attested  to  the  effects  of  mental  contrasting  on  wise  goal 
pursuit  across  a  variety  of  life  domains,  including  scholastic  achievement  (Gollwitzer,  Oettingen, 
Kirby,  &  Duckworth,  2011),  health  (Armitrage,  2004;  Sheeran,  Harris,  Vaughan,  Oetttingen,  & 
Gollwitzer,  2012),  and  interpersonal  relationships  (Oettingen,  Stephens,  Mayer,  &  Brinkmann, 
2010).  In  addition,  the  effects  of  mental  contrasting  have  been  efficacious  across  the  life  span 
(Oettingen,  2000),  across  socioeconomic  statuses  and  cultures  (Annitage,  2004;  Sheeran,  Harris, 
Vaughan,  Oetttingen,  &  Gollwitzer,  2012),  for  short-term  and  long-tenn  goal  pursuits  (Oettingen 
&  Gollwitzer,  2010),  and  for  the  pursuit  of  approach  goals  as  well  as  avoidance  goals  (Houssais, 
Oettingen,  &  Mayer,  2013).  The  effects  have  been  replicated  for  a  wide  variety  of  indicators  of 
goal  pursuit  (e.g.,  cognitive,  emotional,  and  behavioral). 

Selective  pursuit  of  desired  outcomes  is  only  a  first  step  to  successful  goal  attainment 
(Gollwitzer  &  Oettingen,  2012).  Especially  when  goal  pursuit  becomes  highly  complex,  people 
often  fail  to  get  started  (i.e.,  procrastinate),  fail  to  stay  on  track  (i.e.,  get  distracted),  fail  to  cease 


1  See  http://csf2.army.miF  for  additional  information  on  these  programs. 
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futile  goal  striving  (i.e.,  overcommit),  and  fail  to  prevent  overextension  (i.e.,  do  too  much). 
Successful  goal  attainment  may  not  only  require  individuals  to  select  goals  that  are  both 
desirable  and  feasible,  but  also  to  cope  effectively  with  a  number  of  challenges  on  the  way  to 
goal  attainment. 

One  strategy  that  has  been  shown  to  help  individuals  take  control  of  goal  implementation 
is  planning  out  one’s  goal  striving  in  advance  using  if-then  plans  (i.e.,  implementation  intentions, 
Gollwitzer  &  Brandstatter,  1997;  Gollwitzer,  1999).  Implementation  intentions  define  when, 
where,  and  how  one  will  act  on  one’s  goals.  Implementation  intentions  have  the  structure  of  “If 
situation  X  is  encountered,  then  I  will  perform  the  goal-directed  response  Y,”  thereby  spelling 
out  how  the  goal  will  be  realized  once  a  relevant  situational  cue  is  encountered. 

In  recent  years,  the  strategy  of  mental  contrasting  (MC)  has  been  combined  with 
implementation  intentions  (II)  in  one  intervention  exercise:  Mental  Contrasting  with 
Implementation  Intentions  (MCII).  Research  has  indicated  that  MCII  improves  people’s  daily 
lives  and  long-tenn  development  by  helping  them  achieve  their  behavior  change  goals 
(Oettingen,  2012;  Oettingen  &  Gollwitzer,  2010).  Moreover,  MCII  has  been  found  to  produce 
immediate  and  lasting  behavior  change  (Stadler,  Oettingen,  &  Gollwitzer  2009;  2010).  MCII 
also  produced  greater  goal  achievement  than  did  mental  contrasting  or  implementation  intention 
strategies  employed  alone  (Stadler,  Oettingen,  &  Gollwitzer  2009). 

Traditional  intervention  studies  focus  on  changing  participants’  beliefs  about  the 
desirability  and  feasibility  of  their  behavior  as  well  as  on  changing  how  they  understand  their 
goals  and  their  current  situation.  MCII  differs  by  focusing  on  pursuing  desired  outcomes  in 
wise,  effective,  and  efficient  ways.  MCII  can  be  learned  and  applied  to  a  single  targeted 
outcome  (e.g.,  losing  five  pounds),  but  it  can  also  be  applied  to  a  variety  of  concerns  a  person 
may  have  during  everyday  life.  Finally,  MCII  can  be  taught  and  applied  in  time  and  cost- 
effective  ways.  In  addition  to  these  established  strategies,  the  use  of  visualizations  of  goal 
progress  has  also  been  shown  to  increase  motivation  to  achieve  goals  (Cheema  &  Bagchi,  2011). 

Each  of  these  goal  pursuit  strategies  was  included  in  the  current  research  and  examined  as 
they  pertain  to  improving  goal  achievement  through  the  use  of  a  mobile  goal  planning  and 
pursuit  application.  Mobile  technologies  are  of  interest  to  the  Army  given  their  portability  and 
suitability  to  address  training  needs  when  and  where  they  arise  (TRADOC,  2011). 

Method 

Two  mobile  applications  were  developed  to  compare  SMART  and  MCII  goal  planning 
and  pursuit  approaches.  The  following  describes  the  process  of  developing  the  applications  and 
presents  their  key  features.  In  addition,  the  evaluation  process  for  comparing  the  two 
applications  is  presented  and  discussed. 
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Application  Specifications 

Based  on  discussions  with  academics  and  Noncommissioned  Officer  Educational  System 
(NCOES)  instructors,  the  research  team  gathered  recommendations  for  the  best  processes,  steps 
and  content  to  implement  the  SMART  and  MCII  techniques  within  a  smart  phone  application  for 
Soldiers.  These  discussions  included  consideration  of  the  wording,  time  spent,  input 
requirements,  Soldier  thought  processes  and  overall  objectives  related  to  emphasizing  the 
techniques  and  the  utility  of  the  techniques  for  Soldiers  involved  in  specific  goal  planning  and 
pursuit  activities.  These  specifications  were  documented  and  provided  to  the  development  team. 
These  documents  allowed  for  ongoing  discussions  between  the  development  and  research  team 
to  make  improvements  to  the  pages,  functions,  and  layouts  prior  to  initial  development.  The 
applications  were  developed  to  function  equivalently  in  both  iPhone  and  Android  versions. 

Application  Development  Process 

The  application  development  process  for  both  applications  included: 

1 .  Discussions  with  project  team  members  regarding  the  optimal  number  of  screens 
(pages),  screen  content  detail,  input  requirements  and  the  look  of  the  applications  to 
optimize  Soldier  interest,  engagement  and  value  of  using  the  applications. 

2.  Developing  application  content.  Various  content  strategies  were  proposed  for  each  of 
the  screens  for  each  application.  Proposed  content  was  reviewed  by  the  project  team 
multiple  times  to  streamline  what  was  presented  on  each  screen  and  the  input 
requirements  of  the  participants.  A  central  theme  in  developing  content  was  to  make 
the  language  simple  and  straightforward  and  to  limit  the  goal  input  requirements  so 
that  only  a  minimum  number  of  goals  would  need  to  be  set.  This  approach  was 
intended  to  keep  the  use  requirements  straightforward  and  not  too  time-consuming  for 
the  user.  Attempts  were  also  made  to  develop  the  Army  Achieve  and  SMART 
applications  to  equalize  the  time  and  effort  required  to  complete  each  application. 

3.  Developing  two  layout  options,  representing  different  color  schemes  and  icons.  These 
layouts  were  presented  to  Army  Research  Institute  (ARI)  for  review.  Various  logos 
for  both  SMART  and  Army  Achieve  applications  were  considered.  Ultimately,  we 
sought  to  give  the  design  an  Army  flavor  without  making  it  appear  that  the  Army  was 
formally  endorsing  the  applications.  These  logos  appear  in  the  screen  captures  in 
Figures  1-24. 

4.  Developing  representations  of  each  page  in  the  Army  Achieve  (MCII  Application) 
and  the  Army  SMART  (Army  goal  book  based  application)  as  they  would  appear  on 
a  smartphone.  The  code  for  both  applications  was  written  using 
Appcelerator/Titanium2,  a  cross  platform  development  solution  that  uses  Java  Script. 


2  http://www.appcelerator.com 
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5.  Discussions  and  contingency  planning  around  additional  functionality  (e.g., 

programmable  reminders,  random  reminders  on  goal  progress),  limitations  (e.g.,  what 
would  happen  if  the  application  were  open  and  incoming  phone  calls  were  received), 
and  downloading  and  running  the  applications  on  different  phones  and  smartphone 
types  and  applications  (e.g.,  Android,  iOS). 

Army  Achieve  Application 

As  the  purpose  of  the  Army  Achieve  application  was  to  focus  on  the  MCII  principles,  the 
input  requirements  simple  and  short,  limiting  goal  and  other  text  entries  to  60  characters.  3  Android 
and  iPhone  versions  of  the  application  screen  had  some  minor  differences,  primarily  due  to  screen 
size  and  menu  locations.  Screen  captures  from  the  Android  version  of  the  application  are  provided 
in  Appendix  A.  Table  1  describes  the  structure  and  content  of  the  Army  Achieve  application. 

Table  1. 


The  Structure  and  Con  ten  t  of  the  Army  Achieve  Application 


Screen  Identifier 

Purpose/Function 

Data  Entered  by  Participant 

1. 

Home  Screen 

Introduce  the  application 

Participants  entered  profile 
infonnation  on  first  use;  then  used 
screen  to  add  goals. 

2. 

Profile  Information 

Track  user  data 

Course;  Rank;  Field/MOS;  Time  in 
Service 

3. 

Identify  End  of  Course  and  24- 
Hour  Goals 

Have  user  establish  a  near  tenn  and 
long  tenn  goal 

Text  input  describing  goals 

4. 

Identify  Your  Outcome 

Have  user  identify  the  outcomes 
associated  with  their  goals. 

Text  input  describing  outcome 

5. 

Imagine  the  Outcome 

Have  user  picture  themselves 
achieving  the  outcome  in  order  to 
prepare  them  for  mental  contrasting 

No  data  entered 

6. 

Identify  Your  Obstacle 

Have  user  anticipate  what  could 
interfere  with  their  pursuit  of  their 
goals 

Text  input  describing  obstacle;  focus 
was  on  an  internal  rather  than 
external  obstacle 

7. 

Imagine  this  Obstacle 

Have  user  imagine  obstacle  as  part  of 
the  mental  contrasting  process 

No  data  entered 

8. 

Taking  Action 

Have  user  describe  an  action  they  can 
take  to  overcome  the  obstacle 

Text  input  describing  action 

9. 

Creating  an  If  ...Then. ..Plan 

Have  users  clearly  state  what  they 
will  do  if  they  encounter  the  obstacle; 
motivate  goal  pursuit  process 

Text  input  describing  plan;  repeat 
entry  of  text  to  better  instill  the  plan 
in  memory 

10. 

Goal  Summary  Screen 

Provides  users  with  a  summary  of  the 
information  they  entered 

Prompts  users  to  ‘save’  the  data  they 
entered 

11. 

Goal  Progress  Screen 

Allows  users  to  track  their  progress 
for  each  goal  they  entered 

User  indicates  a  ‘percentage 
completed’  in  their  progress  toward 
their  goal 

12. 

Evaluation  of  Application 

Allows  users  to  provide  feedback  on 
the  application 

Users  answer  a  set  of  Likert  style  and 
open-ended  questions  concerning 
their  experience  using  the  application 

On  completing  the  application,  users  were  presented  with  a  screen  that  allowed  them  to 
provide  feedback  on  their  experiences  using  the  application.  The  evaluation  items,  presented  in 


3  These  design  characteristics  were  recommended  by  G.  Oettingen  and  P.  Gollwitzer  (personal 
communication,  March- April  2013). 
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Table  2  below,  included  nine  items  to  be  rated  using  a  Likert-type  agreement  scale  and  three 
open-ended  questions.  Items  1-3  were  analyzed  as  a  composite  scale  associated  with  the  “Utility 
of  the  Application,”  items  4-6  were  associated  with  “Goal  Approach”  and  items  7-9  were 
associated  with  “Motivation.”  Open  ended  responses  were  also  analyzed  for  common  themes. 

Table  2 

Evaluation  Items  and  Scales _ 

Item  Scale 


1.  I  would  recommend  this  application  to  other  NCOs. 


2.  The  application  was  easy  to  use. 

3.  I  would  like  to  continue  to  use  the  application  in  the  future. 

4.  The  application  changed  my  way  of  approaching  challenges. 

5.  The  application  helped  me  more  clearly  identify  my  course 
goals. 

6.  The  application  helped  prioritize  my  learning  tasks. 

7.  The  application  helped  me  learn  more  in  this  course. 

8.  The  application  helped  me  be  more  involved  in  this  course. 

9.  The  application  helped  me  stay  motivated  to  pursue  my  goals. 

10.  The  best  thing  about  the  application  is: 

11.  Something  that  could  be  improved  about  the  application  is: 

12.  Other  comments: 


5-point  agreement  scale  (Strongly  Agree 
(SA),  Agree  (A),  Neither  Agree  nor 
Disagree  (N),  Disagree  (D),  Strongly 
Disagree  (SD) 

5-point  agreement  scale 
5-point  agreement  scale 
5-point  agreement  scale 
5-point  agreement  scale 

5-point  agreement  scale 
5-point  agreement  scale 
5-point  agreement  scale 
5-point  agreement  scale 
Open-ended  response 
Open-ended  response 
Open-ended  response 


A  partial  depiction  of  the  evaluation  screen  in  the  Army  Achieve  application  appears  as  Figure  1. 
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EVALUATION 


MY  EVALUATION 


Please  respond  to  each  question  and 
provide  any  additional  written 
responses  in  the  spaces  provided. 

Use  the  provided  scale: 

5D  =  Strongly  Disagree 
D  =  Disagree 
N  =  Neutral 
A  =  Agree 

SA  =  Strongly  agree. 

I  would  recommend  this  application  to 
other  NCOs. 

SD  D  N  A  SA 


The  application  was  easy  to  use. 

SD  D  N  A  SA 


I  would  like  to  continue  to  use  the 
application  in  the  future. 

SD  D  N  A  SA 


+  =  ±  6  « 

Add  Goal  My  Goals  Profil©  Evaluation  Contac 


Figure  1 .  Partial  Depiction  of  Army  Achieve  Evaluation  Screen. 

After  completing  all  items  in  the  evaluation,  the  user  could  then  click  on  the  “Send” 
button  at  the  bottom  of  the  evaluation  and  an  email  message  would  be  opened  with  the  Army 
Evaluation  Team  address  populated  in  the  send  line.  Users  could  then  send  the  email  directly 
from  any  email  return  address  they  wished  to  use. 

There  was  also  a  means  by  which  users  could  contact  the  Evaluation  Team  with  any 
problems  or  questions  they  had  regarding  the  use  of  the  application  or  completing  the  evaluation. 
A  contact  menu  button  took  users  to  an  open  email  message,  prepopulated  with  the  Evaluation 
Team  email  address.  The  users  could  type  in  any  message  and  send  the  message  from  their 
chosen  email  address. 
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Army  SMART  Application 


The  purpose  of  the  Army  SMART  application  was  to  focus  a  goal  setting  and  pursuit 
process  that  reflected  the  current  Army  Resiliency  program  goal  setting  process.  This  process, 
based  upon  the  Anny  Goals  Book  (Department  of  the  Anny,  Army  Center  for  Enhanced 
Performance,  n.d.),  was  considerably  more  complex  than  the  process  developed  for  the  Army 
Achieve  application  and  was  therefore  modified  somewhat  to  provide  a  comparable  application 
for  evaluation  purposes.  This  section  describes  the  process  and  screens  developed  in  the  Army 
SMART  application.  Android  and  iPhone  versions  of  the  application  screen  had  some  slight 
differences,  primarily  due  to  screen  size  and  menu  locations.  Android  versions  of  the  screen 
captures  for  each  screen  are  provided  in  this  section.  The  various  screens  are  presented  in 
Appendix  B.  Table  3  summarizes  the  structure  and  content  of  the  Army  SMART  application. 

Table  3 


The  Structure  and  Content  of  the  Army  SMART  Application 


Screen  Identifier 

Purpose/Function 

Data  Entered  by  Participant 

1. 

Home  Screen 

Introduce  the  application 

Participants  entered  profile 
information  on  first  use;  then  used 
screen  to  add  goals. 

2. 

Profile  Information 

To  track  user  data 

Drop  down  menus  to  for 
participants  to  enter  data  on  their 
Course,  Rank,  MOS,  etc. 

3. 

Add  Goal 

Identify  a  long-term  goal 

Text  data;  users  allowed  120 
characters  as  opposed  to  the  60 
characters  in  Army  Achieve  app 

4. 

Strengths 

Identify  key  strengths,  skills,  and 
qualities  to  help  users  reach  their 
goals;  users  also  prompted  to  enter 
key  obstacles  and  challenges  to 
reaching  their  goal 

Text  data  (120  characters) 

5. 

Priority 

Identify  something  about 
themselves  to  work  on  to  help  them 
achieve  their  goal 

Text  data  (120  characters) 

6. 

Action/Belief 

Identify  a  key  action  that  is 
specific,  measurable,  attainable, 
realistic,  and  time-bound;  in 
addition,  users  were  asked  to  enter 
a  motivating  belief 

Text  data  (120  characters) 

7. 

Short  Term  Goals 

Identify  a  24-hour  and  7-day  goal 
associated  with  outcome  goal 

Text  data  (120  characters) 

8. 

Mindful 

Identify  what  users  should  do  to 
remain  mindful  of  their  goals 

Text  data  (120  characters) 

9. 

Goal  Summary 

Summarizes  users’  entries,  allows 
them  to  save  their  goals 

Users  save  their  completed  goals 
after  review 

10. 

Goal  Progress  Screen 

Allows  users  to  track  their  progress 
for  each  goal  they  entered 

User  indicates  a  ‘percentage 
completed’  in  their  progress  toward 
their  goal 

11. 

Evaluation  of  Application 

Allows  users  to  provide  feedback 
on  the  application 

Users  answer  a  set  of  Likert  style 
and  open-ended  questions 
concerning  their  experience  using 
the  application 
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The  evaluation  portion  of  the  application  was  identical  to  the  Anny  Achieve  application 
in  that  users  could  access  the  My  Goals  screen  to  indicate  progress  on  goals,  and  access  the 
Evaluation  screen  to  complete  the  application  evaluation  and  email  their  results  to  the  Research 
Team.  In  addition,  users  could  contact  the  research  team  through  the  Contact  screen. 

Application  Fielding  and  Testing 

Following  approval  of  wireframe  content,  the  functional  prototypes  of  the  Anny  Achieve 
application  in  iOS  and  Android  versions  were  developed.  These  versions  were  distributed  to 
team  members  and  ARI  through  TestFlight.4  TestFlight  is  a  developer’s  tool  which  allows 
licensed  users  to  download  test  applications  not  available  to  the  general  public.  Due  to 
differences  in  screen  and  layout  options  on  iPhone  and  Android  systems,  multiple  versions  of  the 
applications  were  necessary  to  properly  represent  application  content  across  the  Profile,  Goal 
Planning,  Goal  Progress  Tracking  and  Application  Evaluation  functions.  Final  applications  were 
distributed  to  participants  through  the  iTunes  ICF  and  Android  ICF  stores  to  limit  the  awareness 
of  the  general  public  as  to  the  availability  of  the  applications. 

Evaluation  of  the  Mobile  Applications 

Participants.  Participants  in  the  Army  Achieve  and  SMART  application  evaluation 
included  students  in  three  NCO  professional  development  courses  held  at  Fort  Feonard  Wood, 
MO.  Students  from  the  Advanced  Reader  Courses  (AFC)  in  Chemical,  Biological,  Radiological, 
and  Nuclear  (CBRN),  Military  Police  (MP),  and  Engineering  (ENG)  participated.  A  total  of  1 16 
NCOs  each  attended  one  of  six  sessions.  A  total  of  1 13  participants  were  eligible,  on  the  basis  of 
the  operating  system  of  their  phones,  and  agreed  to  participate  in  the  research.  The  exact 
breakdown  of  iPhone  to  Android  users  could  not  be  detennined  as  not  all  infonnation  on  the 
sign-in  sheets  was  completed  by  all  participants.  However,  for  those  individuals  who  did 
indicate  their  phone’s  operating  system,  3 1  out  of  66  had  iPhones  and  the  remainder  had 
Androids.  The  three  who  could  not  participate  were  using  Windows-based  phones  or  older  flip- 
phones. 


Materials.  The  Anny  Achieve  application  contained  an  internal  database  that  tracked: 

•  Goal  description  entered,  time  and  date  of  entry, 

•  Type  of  goal  entered  (24-hour,  End  of  Course,  Other), 

•  Outcome  the  student  associated  with  the  goal, 

•  Obstacle  the  student  associated  with  achieving  the  goal, 

•  Action  the  student  identified  with  overcoming  the  obstacle, 

•  If. .  .Then. .  .Plan  the  student  created  to  help  overcome  obstacle,  and 

•  Progress  indicated  on  the  goal. 

The  Army  SMART  application  contained  an  internal  database  that  tracked: 

•  Fong  term  goal  description  entered,  time  and  date  of  entry, 


4  The  testflight  application  software  can  be  downloaded  at  https://testflightapp.coin/ 
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•  Strength,  skill  or  other  quality  identified  to  help  reach  goal, 

•  Obstacle  or  challenge  the  student  associated  with  achieving  the  goal, 

•  Priority  identified  for  self-development, 

•  Key  action  identified, 

•  Key  belief  identified, 

•  Short  tenn  goals  identified  to  reach  long  term  goal, 

•  Description  of  what  the  student  will  do  to  be  mindful  of  the  goal,  and 

•  Progress  achieved  on  the  long  tenn  and  short  term  goals. 

A  student  evaluation  of  the  application  was  also  self-contained  in  the  application.  The 
evaluation  was  identical  for  both  Army  Achieve  and  Anny  SMART  applications.  Students  were 
instructed  to  fill  out  the  evaluation  no  earlier  than  two  days  before  the  end  of  the  course.  After 
students  completed  the  evaluation,  they  could  e-mail  the  evaluation  directly  to  the  study  team. 

Procedure.  Students  were  introduced  to  the  application  during  a  20-45  minute 
orientation  session  that  occurred  at  different  points  in  each  course.  An  attempt  was  made  to 
schedule  this  orientation  for  the  second  week  of  each  course,  but  ultimately  the  orientation 
occurred  between  the  2nd  week  and  next  to  last  week  of  each  course.  During  this  orientation,  the 
following  topics  were  presented: 

•  Privacy  Act  and  Informed  Consent 

•  Objective  of  using  the  application 

•  Application  Installation 

•  Instructions  for  providing  Profile  information,  using  the  application  and  providing  an 
evaluation  of  the  application,  and 

•  Additional  infonnation  regarding  the  evaluation  and  timeframe  for  response. 

The  orientation  was  conducted  in  a  room  with  open  access  wireless  capability. 
Participants  were  provided  access  to  download  the  applications.  They  were  aware  that  there 
were  two  goal  pursuit  strategies  being  tested  with  the  applications.  Each  of  the  six  orientation 
sessions  focused  on  either  the  SMART  or  MCII  applications.  That  said,  the  participants  were 
offered  the  opportunity  to  download  the  other,  either  SMART  or  MCII,  application  on  request 
after  completing  the  research. 

Follow  Up.  Due  to  a  low  response  rate,  participants  who  had  signed  up  for  the 
evaluation  were  sent  two  email  reminders  regarding  completing  the  application  and  sending  the 
results  to  the  research  team.  The  first  reminder  was  sent  at  the  end  of  the  final  week  of  class  for 
the  ALC  course  that  was  the  last  to  finish.  The  second  reminder  was  sent  the  following  week. 
Only  one  additional  evaluation  was  returned  in  response  to  these  reminders. 

Analysis  Strategy.  The  intent  of  the  evaluation  was  to  conduct  primary  analyses 
concentrating  on  descriptive  statistics,  t-tests  and  analysis  of  variance  related  to  the  number  of 
goals  entered,  progress  indicators  related  to  those  goals  and  to  compare  the  student  evaluations 
of  both  goal  planning  applications  across  the  individual  evaluation  items  and  composites. 
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Results 


A  total  of  nine  participants  returned  their  evaluations  using  the  built-in  email  evaluation 
function.  Five  of  the  evaluations  were  submitted  within  three  days  of  the  orientation  session, 
which  was  well  in  advance  of  the  request  to  submit  the  evaluation  just  prior  to  or  just  after  the 
end  of  their  course.  The  summary  evaluation  results,  expressed  as  mean  scores  on  each 
evaluation  item  for  the  Achieve  and  SMART  application  respondents,  appear  in  Table  4. 

Table  4 

Responses  to  User  Evaluation  for  Army  Achieve  and  Army  SMART  Applications 


Item 

Achieve  Mean 
(N=4) 

SMART  Mean 
(N=5) 

1  would  recommend  this  application  to 
other  NCOs 

3.25 

3.60 

The  application  was  easy  to  use 

4.75 

4.20 

1  would  like  to  continue  to  use  the 
application  in  the  future 

3.50 

3.40 

The  application  changed  my  way  of 
approaching  challenges 

2.75 

3.20 

The  application  helped  me  more  clearly 
identify  my  course  goals 

3.75 

3.40 

The  application  helped  prioritize  my 
learning  tasks 

3.25 

3.40 

The  application  helped  me  leam  more  in 
this  course 

2.75 

3.20 

The  application  helped  me  be  more 
involved  in  this  course. 

3.25 

3.20 

The  application  helped  me  stay  motivated 
to  pursue  my  goals 

3.75 

3.40 

Total 

3.44 

3.44 

Note:  Standard  Likert  scale  response  categories  were  used  for  these  items:  1  (Strongly  Disagree),  2  (Disagree),  3 
(Neither  Agree  or  Disagree),  4  (Agree),  and  5  (Strongly  Agree). 

Additional  open-ended  comments  were  solicited  in  regard  to  each  application.  No  firm 
conclusions  could  be  drawn  from  the  comments  provided.  The  Army  Achieve  application  was 
associated  with  slightly  more  comments  overall  than  the  Anny  SMART  application. 

Army  Achieve  Comments 

Participants  described  the  application  as  easy  to  use  and  helpful  to  keep  them  on  track 
with  their  goals.  One  participant  noted  that  the  application  was  good,  but  the  concept  was 
flawed.  Another  noted  the  process  involved  too  much  redundancy.  Finally,  one  participant 
commented  that  the  application  “could  use  more  flare.” 

Army  SMART  Comments 

There  were  very  few  comments  for  the  Anny  SMART  application.  One  participant  noted 
that  the  application  was  easy  to  use,  and  another  that  it  could  have  been  more  colorful. 
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Goals  Entered  into  the  Applications 

The  goals  entered  in  the  Anny  Achieve  application  tended  not  to  be  in-depth,  e.g.,  “make 
Commandant’s  list,”  and  “graduate  with  an  88%  or  higher.”  The  Anny  SMART  goals  tended  to 
be  more  in-depth,  e.g.,  “I  would  like  to  make  commandants  list.  Possibly  be  honor  grad,  help 
distinguish  myself  from  my  peers,”  “I  want  to  earn  commandant  and  leam  more  about  what  it 
takes  to  be  a  leader.”  The  length  of  the  responses  may  have  been  related  to  the  respective 
character  lengths  allowed  in  each  application:  60  characters  for  Anny  Achieve  and  120 
characters  for  Army  SMART.  Three  Anny  Achieve  and  four  SMART  application  users  returned 
their  evaluations  with  goal  data  included. 

Based  upon  the  extremely  low  response  rate  (8%),  no  statistical  tests  were  conducted. 

The  recommendations  section  of  this  report  will  therefore  concentrate  on  the  limited  findings 
from  the  returned  evaluations  and  participant  comments  during  the  larger,  more  representative 
orientation  sessions. 


Discussion 

The  mobile  applications  were  designed  to  provide  Soldiers  with  templates  to  guide  their 
goal  planning  and  pursuit  processes;  the  associated  research  was  focused  on  comparing  two  goal 
planning  and  pursuit  processes.  While  goal  planning  and  pursuit  techniques  are  supported  by 
and  included  in  Army  doctrine  (e.g.,  the  Military  Decision  Making  Process,  and  SMART  goals 
approach),  it  is  unknown  how  frequently  structured  goal  setting  techniques  are  employed  by 
Soldiers.  Previous  research  has  suggested,  however,  that  a  variety  of  techniques  related  to  goal 
planning  and  pursuit  are  employed  by  Anny  NCOs  in  the  context  of  self-directed  learning 
(Graves,  Rauchfuss,  &  Wisecarver,  2011). 

Structured  goal  planning  and  pursuit  can  be  a  valuable  activity,  but  its  value  may  depend 
on  individual  Soldiers’  experience  level,  the  context  of  the  goals  being  pursued,  and  the 
motivation  of  individual  Soldiers  to  use  the  techniques.  Moreover,  certain  types  of  goals  warrant 
an  in-depth  planning  process,  whereas  others  do  not.  For  individuals  who  are  used  to  a  more 
informal,  open-ended  goal  planning  and  pursuit  process,  the  structured  approach  may  not  have 
been  viewed  as  flexible  or  adaptable  enough  to  meet  their  needs.  Unfortunately,  the  low 
response  rate  did  not  allow  us  to  determine  whether  or  not  NCOs  had  a  preference  for  either  of 
the  goal  planning  and  pursuit  techniques.  Although  the  experiment  comparing  the  two 
approaches  to  goal  planning  and  pursuit  was  unsuccessful  due  to  a  low  response  rate,  there  were 
a  number  of  takeaway  points  concerning  the  difficulties  that  may  be  encountered  developing  and 
deploying  digital  applications  that  we  were  able  to  derive  from  our  discussions  with  participants 
in  the  orientation  sessions. 

Selection  of  the  Modality  for  the  Training 

The  goal  planning  and  pursuit  subject  content  did  not  lend  itself  to  the  limited  visual 
format  of  mobile  phones.  With  a  limited  amount  of  space  and  small  font  size,  users  of  the 
applications  may  have  been  challenged  to  enter  their  goals  and  to  respond  to  the  various 
questions  asked  by  the  application  when  walking  them  through  the  goal  planning  process. 
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Limiting  the  number  of  characters  to  keep  all  the  text  on  screen,  and  to  conform  to  the 
requirements  of  the  goal  planning  and  pursuit  techniques  being  evaluated,  likely  did  not  help 
reduce  cognitive  load  associated  with  the  task.  Participants  were  forced  to  be  as  concise  as 
possible  in  responding  to  the  on-screen  questions.  That  said,  this  point  could  be  made 
concerning  other  types  of  applications  the  Army  may  seek  to  develop  for  mobile  devices.  While 
mobile  devices  have  the  advantage  of  almost  always  being  with  Soldiers,  the  devices  are 
necessarily  limited  as  training  technologies  by  their  physical  size  and  storage  capacity.  The  size 
and  capacity  of  the  device  used  for  training  should  always  be  considered  in  the  content  selection 
and  design  phases  of  application  development  (Blankenbeckler,  Graves,  &  Wampler,  2013). 

Individual  Cost/Benefit  Determinations 

While  the  applications  were  made  available  to  participants  for  free,  the  participants  were 
nonetheless  being  asked  to  run  these  applications  on  their  personally-owned  mobile  phones.  As 
storage  space  is  limited  on  these  devices,  users  may  regularly  cull  the  applications  on  their 
phones.  If  an  application  remains  largely  unused,  it  may  be  deleted.  While  participants 
recognized  the  need  for  establishing  good  goal  planning  and  pursuit  processes,  our  applications 
may  not  have  been  viewed  as  facilitating  their  approach  to  that  process.  This  was  suggested  in 
some  participants’  comments  during  the  orientation  sessions  that  “the  concept  is  good,  but  the 
execution  was  flawed.”  If  the  applications  were  not  perceived  to  streamline  the  process  of  goal 
planning  and  pursuit,  then  participants  would  not  have  been  motivated  to  retain  them  on  their 
personal  mobile  phones  for  the  duration  of  the  evaluation  period. 

Delivery  and  Accessibility 

Another  issue  involving  the  use  of  participants’  personally-owned  mobile  phones 
concerned  minimizing  costs  to  them  as  much  as  possible.  In  order  to  avoid  asking  participants  to 
download  the  mobile  applications  against  their  own  data  plans,  we  utilized  the  wireless  network 
at  the  Fort  Leonard  Wood  NCOA.  This  required  a  classroom  in  which  a  wireless  local  area 
network  could  provide  a  clear  signal  that  users  could  access,  and  additional  coordination  with  the 
local  information  technology  specialists  to  grant  the  required  access.  The  wireless  network  at 
Fort  Leonard  Wood  had  sufficient  bandwidth  and  signal  strength  such  that  the  participants  did 
not  need  to  use  their  own  accounts  for  downloading.  Participants  were  able  to  download  the 
applications  in  real  time  during  the  training  sessions.  In  order  for  an  experiment  of  this  type  to 
be  successful,  the  infrastructure  to  distribute  applications  digitally  must  be  in  place.  If  the 
researcher  is  providing  the  devices  to  participants,  rather  than  asking  them  to  volunteer  to  use 
their  personal  devices,  the  devices  could  be  preloaded  with  the  necessary  software.  This, 
however,  would  have  been  a  cost  prohibitive  option  to  pursue  in  order  to  test  simple  goal 
planning  and  pursuit  techniques. 

The  Population  Selected  for  the  Evaluation 

Mid-level  Army  NCOs  may  not  have  been  the  ideal  population  from  which  to  draw  a 
sample  for  the  evaluation.  At  this  point  in  their  careers,  many  of  these  NCOs  have  likely 
developed  preferred  approaches  to  goal  planning  and  pursuit,  having  had  a  number  of  jobs  within 
the  Army  and  a  more  extensive  military  and  civilian  educational  background  (see  Graves, 


13 


Rauchfuss,  &  Wisecarver,  2011).  An  alternative  audience  who  may  benefit  from  exploring 
different  goal  planning  and  pursuit  processes  could  be  early-career  Sergeants  (E-5), 
Specialists/Corporals  (E-4),  and  below. 

Need  for  Integration  with  Existing  Army  Systems 

In  one  orientation  session,  an  NCO  said  “no  one  will  use  the  application”  as  it  does  not 
immediately  appear  to  add  value  for  the  Soldier.  He  felt  there  would  be  value  added  by 
integrating  the  applications  with  the  existing  Anny  Career  Tracker.  In  their  current  state,  the 
applications  were  viewed  as  being  too  simplistic  to  be  compelling.  While  there  was  some 
agreement  in  the  room  with  respect  to  these  comments,  the  researchers  responded  by  indicating 
that  the  evaluation  was  to  determine  the  potential  value  of  the  applications  and  to  compare  the 
respective  approaches  to  goal  planning  and  pursuit.  Later  integration  of  the  applications  with 
larger  Anny  systems  could  be  possible. 

Other  Potential  Reasons  for  Low  Response  Rates 

Participants  were  made  aware  that  their  participation  was  voluntary  and  they  were  not 
under  any  obligation  to  use  the  application  or  to  return  the  evaluation.  Throughout  the 
orientation  sessions,  the  research  team  acknowledged  that  participant  time  was  limited, 
appreciated,  and  integral  to  the  success  of  evaluating  the  mobile  applications.  Participants  may 
have  simply  chosen  not  to  use  the  applications  because  they  did  not  perceive  value  or  a  need  to 
provide  feedback.  During  the  orientation  sessions,  we  sought  to  incentivize  participation  by 
providing  the  participants  with  a  free  mobile  application  that  they  could  keep  if  they  found  it 
useful.  We  also  gave  them  the  option  of  downloading  the  alternate  version  of  the  application 
after  the  evaluation  period. 

Too  much  time  may  have  elapsed  between  the  initial  sessions  and  the  user  evaluation. 
While  participants  were  sent  reminders  to  solicit  their  user  evaluations,  this  resulted  in  very  few 
additional  responses.  Likely,  at  the  end  of  the  course  and  close  of  the  evaluation  period,  the 
NCOs  were  focused  on  returning  to  their  units  and  homes. 

Recommendations 

Recommendations  for  improving  the  evaluation  of  mobile  applications  in  Soldier  samples 
are: 

1 .  Attempt  to  conduct  evaluations  completely  onsite,  with  Soldiers  present  in  the  evaluation 
conditions.  Though  this  is  a  challenging  solution  for  any  application  that  has  value  over  a 
time  period  beyond  the  “test”  condition,  it  seemingly  is  the  best  means  to  ensure 
completion  rates  when  testing  mobile  applications.  Unfortunately,  for  the  present 
research,  the  effectiveness  of  goal  planning  and  pursuit  strategies  are  unlikely  to  be  able 
to  be  tested  over  a  session-length  time  period  (1  to  2  hours)  using  mobile  applications. 
Attrition  can  become  an  issue  in  any  longitudinal  data  collection  (Ahem  &  Le  Brocque, 
2005);  for  this  type  of  evaluation  we  recommend  attempting  to  collect  as  much  data  as 
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possible  in  single  face-to-face  sessions  to  ensure  a  reasonable  sample  size  and  improve 
generalizability. 

2.  Perfonn  pilot  tests  of  the  application  on  groups  that  are  similar  in  characteristics  and 
context  to  those  groups  targeted  for  the  evaluation.  For  the  present  study,  time  did  not 
pennit  for  a  pilot  test  of  the  applications  to  determine  possible  return  rates  in  a  larger 
sample,  or  potential  causes  of  attrition. 

Participants  in  the  mobile  application  goal  setting  orientation  sessions  provided  several 
comments  related  to  desirable  improvements  to  the  applications  prior  to  their  actually  using  the 
application  and  completing  their  evaluations.  These  were  related  to: 

1 .  Linking  the  application  to  existing  Army  planning  and  scheduling  systems,  particularly 
Anny  Career  Tracker. 

2.  Providing  reminders  or  scheduling  function  that  might  prompt  the  user  regarding  subgoal 
steps  or  progress  (i.e.,  integrating  the  application  with  the  users’  calendar). 

The  development  team  did  not  consider  integrating  the  applications  with  any  external 
Anny  systems,  but  rather  sought  to  specifically  evaluate  to  two  goal  planning  and  pursuit 
techniques  in  isolation  from  any  existing  systems.  One  major  concern  was  participants’  privacy, 
so  we  limited  the  integration  of  the  experimental  applications  with  personalized  features  of 
participants’  phones,  such  as  calendars,  address  books,  etc.  The  development  team  did  consider 
providing  reminders  or  a  scheduling  function  but  decided  against  this.  Input  from  MCII  subject 
matter  experts  did  not  recommend  application  reminders  as  these  would  impose  more  rigor  or 
structure  than  is  necessary.  From  the  MCII  perspective,  the  strength  of  the  technique  is  its 
simplicity  of  goal  and  obstacle  imagery  and  the  imprinting  of  the  If. .  .Then. .  .Plan.  Providing  a 
schedule  or  reminders  was  not  recommended  as  it  could  increase  complexity  and  create  obstacles 
to  goal  pursuit. 

Conclusions 

While  goal  planning  and  pursuit  techniques  are  significant  to  the  Army,  a  clean 
evaluation  of  the  two  goal  planning  and  pursuit  processes  was  not  possible  due  to  attrition  among 
the  participants  during  the  evaluation  period.  On  the  basis  of  the  few  responses  received,  no 
differences  were  indicated  between  the  two  goal  planning  and  pursuit  techniques. 
Recommendations  for  future  research  using  mobile  applications  include  exploring  the  possibility 
of  better  integrating  the  application  with  existing  Anny  systems  and  to  focus  the  evaluation  on 
single  session,  face-to-face  data  collections. 

As  the  U.S.  Anny  continues  to  focus  on  developing  Soldiers’  cognitive  skills  and 
enhancing  their  learning  capabilities,  training  and  job  aids  to  support  adaptability,  decision¬ 
making  in  complex  situations,  etc.,  will  be  in  demand  (Department  of  the  Anny,  Maneuver 
Center  of  Excellence,  2012).  Mobile  devices  are  an  obvious  technology  for  distributing  and 
utilizing  training  and  tools  focused  on  cognitive  skills  development:  these  devices  are  in 
widespread  use,  are  highly  portable,  and  can  access  large  amounts  of  data  through  wireless 
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networks.  However,  these  devices  also  have  their  limitations,  including  the  physical  dimensions 
of  their  screens,  their  cost,  and  the  significant  need  to  protect  participants’  privacy,  particularly 
when  asking  participants  to  volunteer  to  use  their  personally  owned  devices.  Any  application  of 
mobile  technology  will  elicit  key  questions  concerning  how  to  optimize  the  design  of  training 
and  tools  for  mobile  devices  as  well  as  how  to  appropriately  integrate  these  technologies  with 
existing  Anny  systems,  while  maintaining  vigilance  regarding  the  Anny’s  infonnation  security 
requirements  and  protecting  Soldiers’  private  information  (Department  of  the  Anny,  Training 
and  Doctrine  Command,  2015). 
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APPENDIX  A 

Screen  Shots  and  Descriptions  for  the  Anny  Achieve  Application 
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Appendix  A:  Army  Achieve  Application 


As  the  purpose  of  the  Army  Achieve  application  was  to  focus  on  the  MCII  principles, 
Professors  Oettingen  and  Gollwitzer  (personal  communication,  March-April  2013)  provided 
extensive  input  concerning  the  content  of  the  application.  They  stressed  keeping  the  input 
requirements  simple  and  short,  limiting  goal  and  other  text  entries  to  60  characters  in  length. 

The  following  sections  describe  the  final  content  included  in  the  Anny  Achieve  application. 
Android  and  iPhone  versions  of  the  application  screen  had  some  minor  differences,  primarily  due 
to  screen  size  and  menu  locations.  Android  versions  of  the  screen  captures  for  each  screen  are 
provided  in  this  section. 

Home  Screen 

This  screen  provided  a  short  introduction  to  the  application  to  encourage  participants  to 
use  the  application.  The  only  navigation  instructions  from  this  screen  were  to  Add  Goal, 
however,  menu  buttons  related  to  My  Goals,  Profile,  Evaluation  and  Contact  were  also  visible. 
Upon  clicking  on  any  of  the  navigation  buttons,  a  pop  up  window  appeared  that  stated  “Please 
enter  your  infonnation  on  the  Profile  tab  before  adding  a  goal”  and  the  user  was  taken  to  the 
Profile  screen  to  complete  Profile  infonnation  upon  selecting  the  “ok”  button.  The  content  for 
the  home  screen  is  presented  as  Figure  A-l. 
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ARMY  ACHIEVE 


PERSONAL  &  PROFESSIONAL 
COAL  SETTING 


This  application  will  help  you 
achieve  your  NCOES  course  goals. 
With  a  few  minutes  of  use,  this 
application  will  guide  you  to 
achieve  challenging,  but  reachable 
goals. 


Add  Goal 


Figure  A-l.  Army  Achieve  Home  Screen. 


Profile  Information 


A  profile  page  was  created  to  track  evaluation  participant  data  (Figure  A-2).  It  consisted 
of  drop  down  menus  and  text  entry  fields  to  make  data  entry  as  user-friendly  as  possible. 
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MY  PROFILE 


MY  PROFILE 

Please  update  your  profile  information 
below 

Course: 

COURSE  V 


Years: 

Months: 


+  =  ±  isa  ® 

Add  Goal  My  Goals  Profile  Evaluation  Contact 


Figure  A -2.  Army  Achieve  Profile  Screen. 

This  page  could  be  accessed  at  any  time  to  update  information  as  desired  by  the  user. 

The  following  fields  and  user  entry  methods  were  included: 

•  Course  (drop  down  menu  with  options  for  Basic  Leader  Course,  Advanced  Leader 
Course,  Senior  Leader  Course), 

•  Rank  (drop  down  menu  with  options  for  SPC/CPL,  SGT,  SSG,  SFC), 

•  Field/MOS  (free  form  text  entry),  and 

•  Time  in  Service  (free  form  entry  into  Years  and  Months  fields). 

For  free  form  entry  fields  on  this  and  all  subsequent  pages  requiring  free  form  text  entry, 
clicking  in  the  free  fonn  field  opened  the  user’s  keyboard  to  enter  text.  The  smartphone’s 
standard  data  editing  and  entry  capabilities  were  enabled  for  text  entry. 

After  entering  all  infonnation  and  clicking  on  the  return  keyboard  button,  a  pop  up 
“Save”  button  appeared.  After  clicking  on  Save,  the  user  was  then  returned  to  the  home  page 
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and  the  “+Add  Goal”  button  opened  a  pop  up  window  stating  “To  begin  using  this  application, 
you  must:  1.  Set  an  End  of  Course  Goal.  2.  Set  a  24-hour  goal.  You  can  then  set  any  additional 
goals  for  any  time  period.”  The  user  then  clicked  on  a  continue  button  to  advance  to  the  Identify 
Your  Goal  screen  (Figure  A-3).  For  this  and  subsequent  screens  requiring  user  input,  a 
“Continue”  button  appeared  at  the  bottom  of  each  page  following  data  entry. 


ARMY  ACHIEVE 

PERSONAL  &  PROFESSIONAL 
COAL  SETTING 


Thlr  ~inn I i r in n  lAiill  hnln  wrm _ 

X 

To  begin  using  this 
application  you  must: 

1.  Set  an  End  of  Course 
Coal. 

2.  Set  a  24-hour  goal. 

You  can  then  set  any 
additional  goals  for  any 
time  period. 


Continue 


Figure  A-3.  Advancing  to  the  Identify  Your  Goal  screen  in  the  Army  Achieve  app. 

Add  Goal/Identify  Your  Goal 

After  advancing  to  the  Add  Goal/Identify  Your  Goal  screen,  users  were  instructed  to 
“First,  identify  your  most  important  end  of  course  goal.  This  goal  should  be  challenging,  but 
achievable.”  A  drop  down  menu  appeared,  but  only  the  End  of  Course  goal  could  be  selected  the 
first  time  the  user  encounters  this  screen.  A  text  entry  field  indicating  “Goal  (60  characters  or 
less)”  appeared  in  which  the  participants  could  add  a  short  description  of  their  End  of  Course 
goal.  There  were  no  restrictions  placed  on  what  text  could  be  entered  into  the  field.  Spell  check 
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may  have  been  active,  depending  upon  the  smartphone  being  used  by  the  participants.  In 
addition,  some  phones  would  also  allow  for  the  participant  to  select  the  microphone  option  and 
speak  their  entry  which  would  then  be  translated  to  text  by  their  phones  speech  to  text  translation 
application.  Figure  A-4  depicts  to  the  Add  Goal/Identify  Your  Goal  screen  for  the  Army 
Achieve  application. 


Figure  A-4.  Anny  Achieve  Add  Goal/Identify  Your  Goal  Screen. 


Identify  Your  Outcome 

The  Identify  Your  Outcome  screen  asked  the  participant  “What  is  the  best  outcome  you 
associate  with  achieving  your  goal?”  This  information  was  deemed  important  to  allow  the 
participant  to  begin  imagining  how  the  goal  differed  from  their  current  situation  in  order  to  build 
value  for  the  participant.  The  user  was  again  provided  with  a  60-character  free  form  entry  field 
to  identify  their  outcome.  Figure  A-5  provides  a  depiction  of  the  Outcome  screen. 
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Figure  A-5.  Anny  Achieve  Identify  Your  Outcome  Screen. 

Clicking  on  the  “Continue”  button  opened  a  pop-up  window  that  read: 

Take  a  minute  or  two  to  imagine  this  outcome.  [The  outcome  the  NCO  entered  is 
displayed.]  Picture  yourself  achieving  this  outcome.  After  you  have  a  clear  image  in 
your  mind,  touch  Continue. 

The  purpose  of  this  exercise  was  to  assist  the  participant  in  mental  contrasting  to 
reinforce  the  importance  of  the  goal  as  well  as  what  it  will  be  like  to  achieve  it.  This  popup 
appears  as  Figure  A-6. 
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f  OUTCOME 

IDENTIFY  YOUR  OUTCOME 


What  is  the  best  outcome  you 
associate  with  achieving  your  goal? 


X 

Take  a  minute  or  two  to 
imagine  this  outcome. 

My  outcome  is... 

Picture  yourself  achieving  this 
outcome.  After  you  have  a  clear  image 
in  your  mind,  touch  "Continue". 


Continue 


Continue  * 


Figure  A-6.  Popup  Screen  Following  Input  of  Participant  Identified  Outcome  of  Achieving 
Goal. 

Identify  Your  Obstacle 

After  imagining  the  outcome  and  clicking  the  continue  button,  users  were  then  instructed 
to  identify  the  main  obstacle  to  achieving  their  goal.  This  screen  instructed  users  to  “Identify 
your  obstacle:  What  stands  in  the  way  of  reaching  your  goal?  (What  is  it  within  you  that  may 
prevent  you  from  reaching  your  goal?)”  To  provide  proper  motivation  for  overcoming  the 
obstacle,  it  was  important  that  the  obstacle  was  expressed  as  something  within  the  participant 
versus  external  to  them.  Again,  a  60-character  free  entry  field  was  provided  to  identify  the 
obstacle  (See  Figure  A-7). 
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Figure  A-7.  Army  Achieve  Identify  Your  Obstacle  Screen. 


After  clicking  on  the  continue  button,  another  popup  window  appeared  that  stated  “Take 
a  minute  or  two  to  imagine  this  obstacle.”  Users  were  again  encouraged  to  imagine  their 
identified  obstacle  as  part  of  the  mental  contrasting  portion  of  the  process.  This  screen  appears 
as  Figure  A-8. 
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^  MAIN  OBSTACLE 

IDENTIFY  YOUR  OBSTACLE 

What  stands  in  the  way  of  reaching 
your  goal?  (What  is  it  within  you  that 


+  S  ±  6  m 


Figure  A-8.  Army  Achieve  Imagining  Obstacle  Screen. 

Taking  Action 

After  users  again  imagined  their  main  obstacle,  they  were  asked  to  identify  an  Action  to 
overcome  the  obstacle.  Figure  A-9  depicts  the  Taking  Action  screen.  Once  again,  users  were 
provided  with  a  free  form,  60-character  text  entry  field  for  identifying  this  action.  Unlike  the 
previous  two  pages,  there  was  no  ‘imagining’  popup  window  that  appeared  when  they  completed 
this  page. 
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^  TAKING  ACTION 

IDENTIFYING  AN  ACTION 

What  action  should  you  take  to 
overcome  this  obstacle? 

ACTION  (60  CHARACTERS) 


Continue  * 


Figure  A-9.  Army  Achieve  Taking  Action  Screen. 

Creating  an  If... Then . . .Plan 

Once  users  had  identified  an  action  to  overcome  the  obstacle,  they  were  asked  to  “make  a 
simple  If. .  .Then. . .  Plan  by  filling  in  your  obstacle  and  the  action  you  will  take  to  overcome  that 
obstacle  in  fields  below.”  The  user  was  prompted  to  re-enter  the  information  related  to  the 
obstacle  and  the  action  to  take  as  depicted  in  Figure  A- 10.  While  this  screen  is  essentially  a 
combination  of  the  two  previous  screen  functions,  it  is  important  in  the  implementation 
intentions  process  in  that  it  links  the  obstacle  to  the  action  identified  to  overcome  that  obstacle. 

It  then  requires  the  user  to  repeat  this  plan  a  second  time.  This  If. .  .Then. .  .Plan  is  critical  in 
maintaining  motivation  to  pursue  goals  and  especially  to  be  better  prepared  to  overcome 
obstacles  on  the  journey  to  goal  achievement.  While  the  application  could  have  “prepopulated” 
the  If. .  .Then. .  .Plan  with  the  entries  from  the  Identify  Obstacle  and  Taking  Action  pages,  it  was 
deemed  important  that  the  user  re-enter  this  information  to  help  them  better  recall  and  implement 
their  plan.  Another  key  aspect  of  this  process  was  to  use  the  phrase  “then  I  will”  with  respect  to 
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what  action  will  be  taken.  This  phrasing  was  felt  to  encourage  users  by  explicitly  stating  what 
they  will  do  to  overcome  the  obstacle. 


^  IF.. .THEN. ..PLAN 

CREATING  AN 
IF.. .THEN. ..PLAN 

Now,  make  a  simple  If. ..Then. ..Plan  by 
filling  in  your  obstacle  and  the  action 
to  overcome  that  obstacle  below. 

If: 

FILL  IN  OBSTACLE 

Then  I  will: 

FILL  IN  ACTION 


Continue  * 


Figure  A-10.  Anny  Achieve  If. .  .Then. .  .Plan  Screen. 


Following  the  initial  entry  of  the  If. . . .  Then. .  .Plan  infonnation,  the  user  was  prompted 
to  repeat  the  process  to  re-enter  the  obstacle  and  action  a  second  time.  This  repetition  of  the  link 
between  the  obstacle  and  the  action  begins  building  memory  for  the  plan.  A  screen  capture 
depiction  of  the  repeating  of  the  If...  Then  ...  Plan  appears  in  Figure  A-l  1. 
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^  IF.. .THEN. ..PLAN 

REPEAT  YOUR 
IF.. .THEN. ..PLAN 


Now,  repeat  your  If.. .Then. ..Plan  by 
retyping  your  obstacle  and  the  action 
to  overcome  that  obstacle.  Repeating 
this  plan  will  help  you  remember  your 
plan. 

If: 


Then  I  will: 


Continue  * 


Figure  A- 1 1 .  Army  Achieve  Repeating  of  the  If. . .  Then. .  .Plan  Screen. 


Following  the  entry  of  the  If. .  .Then. .  .Plan  and  the  repeating  of  the  Plan,  the  user  was 
again  prompted  with  a  popup  window  which  stated  “Take  a  few  moments  to  imagine  your 
If. .  .Then. .  .Plan.”  The  application  populated  the  obstacle  and  action  taken  information  from  the 
previous  If. .  .Then. .  .Plan  screen.  After  users  have  imagined  the  Plan,  they  would  then  click 
continue.  The  imagining  portion  of  the  If. .  .Then. .  .Plan  screen  is  depicted  in  Figure  A-12. 
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REPEAT  YOUR 
IF.. .THEN. ..PLAN 


Now,  repeat  your  If. ..Then. ..Plan  by 
retyping  your  obstacle  and  the  action 
to  overcome  that  obstacle.  Repeating 
this  plan  will  help  you  remember  your 
plan. 

X 

Now,  take  a  few  moments  to 
imagine  your  If-Then  Plan: 

If: 

Then  I  Will: 

After  you  have  a  clear  image  in  your 
mind,  touch  "Continue". 


Continue 


+  =  ±  6  ® 

Add  Goal  My  Goals  Profile  Evaluation  Contact 


Figure  A-12.  Depiction  of  If. .  .Then. .  .Plan  Imagining  Instruction  of  the  Anny  Achieve 
Application. 


Goal  Summary  Screen 

After  the  users  clicked  on  the  Continue  after  they  imagine  the  If. .  .Then. .  .Plan,  the  user 
was  taken  to  the  Summary  screen.  The  Summary  screen  provided  all  information  entered  by  the 
user  related  to  the  previous  screens,  and  consisted  of  the  information  as  depicted  in  Figure  A-13. 
The  purpose  of  the  Summary  page  was  to  provide  a  review  of  the  infonnation  previously 
entered.  There  was  no  edit  function  built  into  the  Summary  page.  The  “Save”  button  took  the 
user  to  the  My  Goals  page  and  allowed  the  user  to  track  their  progress  on  the  goal. 
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«■ 


SUMMARY 


1.  MY  COAL: 

My  goal  is... 

2.  MY  OUTCOME: 

My  outcome  is... 

3.  MY  OBSTACLE: 

My  obstacle  is... 

4.  MY  ACTION: 

My  action  is... 

5.  MY  IF. ..THEN. ..PLAN: 

If  ,  then  . 

Save 


+  =  ±  fi 

Add  Goal  My  Goals  Profile  Evaluation  Contact 

Figure  A- 13.  Army  Achieve  Summary  Screen. 

Following  the  Summary  screen,  users  could  then  track  their  goal  progress  by  clicking  on 
the  My  Goals  icon  in  the  menu  portion  of  the  application.  Doing  so  would  take  them  to  the  My 
Goals  screen  which  would  also  allow  them  to  add  new  goals.  Figure  A- 14  depicts  this  screen. 
To  enter  progress  in  on  existing  goals,  the  user  would  select  the  goal  from  the  list  and  then  click 
on  the  appropriate  percentage  button  related  to  their  progress.  The  system  would  date  and  time 
stamp  any  entries  for  later  analysis  upon  completion  of  the  evaluation. 
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ARMY  ACHIEVE 


MY  COALS 


+  =  X  0  ® 

\dd  Goal  My  Goals  Profile  Evaluation  Contact 


Figure  A- 14.  Army  Achieve  My  Goals  screen. 

Participants  were  asked  to  complete  the  evaluation  after  they  had  finished  using  the 
application.  The  evaluation  items,  presented  in  Table  A-l  below,  included  nine  items  to  be  rated 
using  a  Likert-type  agreement  scale  and  three  open-ended  questions.  Items  1-3  were  analyzed  as 
a  composite  scale  associated  with  the  “Utility  of  the  Application,”  items  4-6  were  associated 
with  “Goal  Approach”  and  items  7-9  were  associated  with  “Motivation.”  Open  ended  responses 
were  also  analyzed  for  common  themes.  A  partial  depiction  of  the  evaluation  screen  in  the  Army 
Achieve  application  appears  as  Figure  A- 15. 
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Table  A-l 

Evaluation  Items  and  Scales 


Item 


Scale 


1 .  I  would  recommend  this  application  to  other  NCOs. 


2.  The  application  was  easy  to  use. 

3.  1  would  like  to  continue  to  use  the  application  in  the 
future. 

4.  The  application  changed  my  way  of  approaching 
challenges. 

5.  The  application  helped  me  more  clearly  identify  my 
course  goals. 

6.  The  application  helped  prioritize  my  learning  tasks. 

7.  The  application  helped  me  learn  more  in  this  course. 

8.  The  application  helped  me  be  more  involved  in  this 
course. 

9.  The  application  helped  me  stay  motivated  to  pursue  my 
goals. 

10.  The  best  thing  about  the  application  is: 

1 1 .  Something  that  could  be  improved  about  the  application 
is: 

12.  Other  comments: 


5-point  agreement  scale  (Strongly 
Agree  (SA),  Agree  (A),  Neither  Agree 
nor  Disagree  (N),  Disagree  (D), 
Strongly  Disagree  (SD) 

5-point  agreement  scale 
5-point  agreement  scale 

5-point  agreement  scale 

5-point  agreement  scale 

5-point  agreement  scale 
5-point  agreement  scale 
5-point  agreement  scale 

5-point  agreement  scale 

Open  response 
Open  response 

Open  response _ 
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EVALUATION 


MY  EVALUATION 


Please  respond  to  each  question  and 
provide  any  additional  written 
responses  in  the  spaces  provided. 

Use  the  provided  scale: 

5D  =  Strongly  Disagree 
D  =  Disagree 
N  =  Neutral 
A  =  Agree 

SA  =  Strongly  agree. 

I  would  recommend  this  application  to 
other  NCOs. 

SD  D  N  A  SA 


The  application  was  easy  to  use. 

SD  D  N  A  SA 


I  would  like  to  continue  to  use  the 
application  in  the  future. 

SD  D  N  A  SA 


+  =  ±  6  « 
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Figure  A-15.  Partial  Depiction  of  Army  Achieve  Evaluation  Screen. 

After  completing  all  items  in  the  evaluation,  the  user  could  then  click  on  the  “Send” 
button  at  the  bottom  of  the  evaluation  and  an  email  message  would  be  opened  with  the  Army 
Evaluation  Team  address  populated  in  the  send  line.  Users  could  then  send  the  email  directly 
from  any  email  return  address  they  wished  to  use. 

There  was  also  a  means  by  which  users  could  contact  the  Evaluation  Team  with  any 
problems  or  questions  they  had  regarding  the  use  of  the  application  or  completing  the  evaluation. 
A  contact  menu  button  took  users  to  an  open  email  message,  prepopulated  with  the  Evaluation 
Team  email  address.  The  users  could  type  in  any  message  and  send  the  message  from  their 
chosen  email  address. 
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APPENDIX  B 

Screen  Shots  and  Descriptions  for  the  Anny  SMART  Application 
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Appendix  B:  Army  SMART  Application 


The  purpose  of  the  Army  SMART  application  was  to  focus  a  goal  setting  and  pursuit 
process  that  reflected  the  current  Anny  Resiliency  program  goal  setting  process.  This  process, 
based  upon  the  Anny  Goals  Book  (Department  of  the  Army,  Army  Center  for  Enhanced 
Perfonnance,  n.d.),  was  considerably  more  complex  than  the  process  developed  for  the  Army 
Achieve  application  and  was  therefore  modified  somewhat  to  provide  a  comparable  application 
for  evaluation  purposes.  This  section  describes  the  process  and  screens  developed  in  the  Army 
SMART  application.  Android  and  iPhone  versions  of  the  application  screen  had  some  slight 
differences,  primarily  due  to  screen  size  and  menu  locations.  Android  versions  of  the  screen 
captures  for  each  screen  are  provided  in  this  section. 

Home  Screen 

This  screen  provided  a  short  introduction  to  the  application  to  encourage  participants  to 
use  the  application.  The  only  navigation  instructions  from  this  screen  were  to  Add  Goal, 
however,  menu  buttons  related  to  My  Goals,  Profile,  Evaluation  and  Contact  were  also  visible. 
Upon  clicking  on  any  of  the  navigation  buttons,  a  pop  up  window  appeared  that  stated  “Please 
enter  your  information  on  the  Profile  tab  before  adding  a  goal”  and  the  user  was  taken  to  the 
Profile  screen  to  complete  Profile  infonnation  upon  selecting  the  “ok”  button.  The  content  for 
the  home  screen  is  presented  as  Figure  B-l. 
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ARMY  SMART 


COAL  SETTING  APPLICATION 

With  a  few  minutes  of  use,  this 
application  will  guide  you  to 
achieve  specific,  measurable, 
attainable,  realistic  and  time- 
bound  goals. 


-|-  Add  Goal 


Figure  B-l.  Army  SMART  Home  Screen. 

Profile  Information 

The  profile  page  was  identical  to  the  Army  Achieve  application.  It  was  created  to  track 
evaluation  participant  data  and  consisted  of  drop  down  menus  and  text  entry  fields  to  make  data 
entry  as  user-friendly  as  possible.  This  page  appears  as  Figure  B-2. 
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MY  PROFILE 


MY  PROFILE 

Please  update  your  profile  information 
below 

Course: 


COURSE  V 


Years: 


Months: 


+  =  1  fi  ® 

Add  Goal  My  Goals  Profile  Evaluation  Contact 

Figure  B-2.  Army  SMART  Profile  Screen. 

The  profile  page  could  be  accessed  at  any  time  to  update  information.  The  following 
fields  and  user  entry  methods  were  included: 

•  Course  (drop  down  menu  with  options  for  Basic  Leader  Course,  Advanced  Leader 
Course,  Senior  Leader  Course) 

•  Rank  (drop  down  menu  with  options  for  SPC/CPL,  SGT,  SSG,  SFC) 

•  Field/MOS  (free  form  text  entry) 

•  Time  in  Service  (free  form  entry  into  Years  and  Months  fields). 

For  free  form  entry  fields  on  this  and  all  subsequent  pages  requiring  free  fonn  text  entry, 
clicking  in  the  free  fonn  entry  portion  of  the  screen  would  open  the  users  keyboard  to  enter  any 
desired  text.  The  smartphone’s  standard  data  editing  and  entry  capabilities  were  enabled  for  text 
entry. 
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Add  Goal 


Once  users  completed  entering  all  profile  information  and  clicking  on  the  return  keyboard 
button,  a  pop  up  “Save”  button  would  appear.  After  clicking  on  Save,  users  were  then  returned 
to  the  home  page  and  the  +Add  Goal  button  would  take  users  to  the  Add  Goal  screen.  They 
would  be  prompted  to  identify  a  long  term  goal  and  enter  the  goal  into  a  free  form  window. 

Users  were  allowed  120  characters  for  entering  goal  information  in  the  SMART  application 
versus  the  60  characters  that  were  available  in  the  Achieve  application.  This  was  done 
intentionally.  The  SMART  goal  process  typically  requires  more  extensive  input  on  the  part  of 
users  compared  to  MCII.  The  Add  Goal  screen  appears  as  Figure  B-3.  Identical  in  function  to 
the  Anny  Achieve  application,  a  “Continue”  button  would  appear  at  the  bottom  of  each  page 
following  entry  of  the  data. 


Figure  B-3.  Anny  SMART  Identify  Your  Long-Term  Goal  Screen. 
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Strengths 


After  the  initial  goal  was  entered,  users  would  be  taken  to  the  Strengths  screen  and 
prompted  to  identify  key  strengths,  skills,  and  qualities  that  will  help  them  reach  their  goal.  This 
screen  appears  as  Figure  B-4.  Users  were  also  prompted  to  enter  key  obstacles  or  challenges  in 
the  way  of  reaching  their  goal. 


STRENGTHS 


Step  2  -  Where  are  you  now? 

Identify  your  key  strengths,  skills,  and 
other  qualities  that  will  help  you  reach 
your  goal. 

Strengths,  Skills  or  Other  qualities: 
(120)  characters) 


Identify  key  obstacles  or  challenges  in 
the  way  of  reaching  your  goal. 

Obstacles  or  challenges:  (120) 
characters) 


Continue  * 


Figure  B-4.  Army  SMART  Application  Strengths  Screen. 

Priority 

The  Priority  screen  came  after  the  Strengths  screen.  It  prompted  users  to  identify 
something  about  themselves  that  they  would  like  to  work  on  to  help  them  reach  their  goal.  The 
Priority  screen  is  depicted  in  Figure  B-5. 
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PRIORITY 


Step  3  -  Identify  a  priority  for 


self^development 


Identify  something  about  yourself  that 
you  would  like  to  work  on  in  order  to 
nelp  you  reach  your  goal. 


Need  to  develop:  (120)  characters) 


Continue 


±  isa 


Figure  B-5.  Priority  Screen  from  the  Army  SMART  Application. 


Action  /Belief 

Following  entry  of  Priority,  users  were  then  taken  to  the  Action  /  Belief  screen  and 
prompted  to  enter  a  key  action  that  is  specific,  measurable,  attainable,  realistic  and  time -bound.  It 
is  on  this  screen  that  the  SMART  goal  planning  elements  are  most  prominent  in  the  application. 
In  addition  to  the  action,  users  were  also  prompted  to  enter  a  powerful  and  personally  motivating 
belief.  The  Action  /  Belief  screen  is  depicted  in  Figure  B-6. 
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^  Action  /  Belief 

Step  4  -  Identify  a  key  action 
ana  belief 

Enter  a  key  action  you  can  take  that  is 
specific,  measurable,  attainable, 
realistic  and  time-bound. 

Action:  (120)  characters) 


Enter  a  key  belief  you  have  that  is 
powerful  and  personal  in  motivating 
you  to  act. 

Belief:  (120)  characters) 


Continue  * 


+  s  ±  fi  » 

Add  Goal  My  Goals  Profile  Evaluation  Contact 


Figure  B-6.  Action  and  Belief  Screen  from  the  Army  SMART  Application. 
Short  Term  Goals 


After  completing  the  Action/Belief  screen,  users  were  taken  to  the  Short  Term  Goals 
screen.  On  this  screen,  users  were  prompted  to  enter  one  24-hour  and  one  7-day  goal  associated 
with  their  outcome  goal.  They  were  further  instructed  that  having  short  term  goals  will  help 
them  reach  their  long  term  goal.  Free  fonn  fields  were  provided  for  the  24-hour  and  7-day  goals. 
This  screen  is  depicted  as  Figure  B-7. 
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^SHORTTERM  COALS 

Step  S  -  Identify  short  term 
goals 

Enter  one  24-hour  and  one  7-day  goal 
associated  with  your  outcome  goal. 
Having  short  term  goals  will  help  you 
reach  your  long  term  goal. 

24-hour  goal:  (120)  characters) 


7-day  goal:  (120)  characters) 


Continue  * 


Figure  B-7.  Army  SMART  Application  Short-Term  Goals  Screen. 

Mindful 


The  final  step  in  the  SMART  process  was  for  users  to  enter  what  they  would  do  that 
would  help  them  be  mindful  of  their  goals.  This  screen  is  depicted  in  Figure  B-8. 
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Figure  B-8.  Army  SMART  Application  Mindful  Screen. 

Goal  Summary 

Following  the  completion  of  the  Mindful  screen,  users  were  taken  to  the  Goal  Summary 
screen.  The  Summary  screen  consisted  of  the  previously  entered  information  as  depicted  in 
Figure  B-9.  The  purpose  of  the  Summary  page  was  to  provide  a  review  of  the  information. 
There  was  no  edit  function  built  into  the  Summary  page.  The  “Save”  button  took  the  user  to  the 
My  Goals  page  and  allowed  the  user  to  track  their  progress  on  the  goal. 
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Figure  B-9.  Army  SMART  Application  Summary  Screen. 

The  remainder  of  the  application  functioned  identically  to  the  Army  Achieve  application 
in  that  users  could  access  the  My  Goals  screen  to  indicate  progress  on  goals,  and  access  the 
Evaluation  screen  to  complete  the  application  evaluation  and  email  their  results  to  the  Research 
Team.  In  addition,  users  could  contact  the  research  team  through  the  Contact  screen. 
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